Measuring the Subjective User eXperience

Measuring the subjective user experience is a challenging task. In this tutorial we will demonstrate how psychological constructs can be divided in separate variables, each measured by its individual questionnaire items. The tutorial will address the analysis of the questionnaire data to estimate its validity and reliability. Analysis will be demonstrated using SPSS. Duration. 3 hours total. 1 Measuring the subjective user experience Recently you could have read the following: “The evaluated prototype led to a higher sense of social connectedness. This was especially true for the 3D interface”. At first glance this seems positive: Social connectedness sounds like something you would want, and if this is provided by the evaluated prototype than perhaps we should pursue developing it as a product. However, several questions remain: 1. What exactly is social connectedness? 2. How is it operationalized in this study? 3. Has it been measured reliably? 4. Validly? And as such is the initial gut feeling of a positive result actually justified? This tutorial addresses just these questions. We will show how psychologists tackle the problems of measuring the subjective user experience. We will address hypothetical constructs (such as social connectedness), variables and items. We will address the different types of validity and reliability. Furthermore, we will address the phrasing and wording of questionnaire items [4]. Finally we will address the quantitative analysis of the results of an administered questionnaire. Common problems associated with using existing questionnaires are also discussed. Questionnaire development starts with the identification of the hypothetical construct one intends to measure, for example social connectedness. After identifying the construct one has to determine which underlying variables would properly reflect social connectedness: bond with your social network, the sense of belonging, the feeling of being in touch... After identifying the variables, these have to be measured using different items: the actual questions shown to participants [2]. Next we focus on validity and reliability. Validity reflects the extent to which your questions measure what you intend to measure. The tutorial first addresses construct validity and then addresses content validity. Furthermore, internal and external validity, ecological validity, temporal validity and face validity are addressed. Reliability reflects the extent to which the measurement reflects the actual score a participant has on the hypothetical construct and thus the extent to which error is omitted from the measurement. We will address test-retest reliability, split-half reliability, and inter-rater reliability [1]. We will continue with a hands-on approach to writing questionnaire items. We will address the informed consent and the special layout of the questionnaire. Furthermore, question wording and inability or unwillingness to answer are addressed. Finally we address the effects of the order of questionnaire items. At this point in the tutorial it should be clear to participants how to start building up a questionnaire based on psychological literature, evaluate the validity and reliability of previously reported questionnaires, and design ones own items and questionnaire. The remainder of the tutorial will focus on two quantitative techniques to analyze the results of the administered questionnaire. First off all we will focus on so-called reliability analysis – the systematic analysis of inter item correlations to be able to conclude whether the phrased items indeed reflect one variable. From the concept of item correlation we proceed to inter-item correlations and to scale analysis. Cronbach’s Alpha [3] is presented as a measure for scale reliability and interpretations are given. Furthermore, examples are provided of methods to increase scale reliability. Analysis will be supported by practical examples using SPSS. Secondly we will look at Principal Component Analysis [5]. The purpose of this technique – to identify latent variables in an item set – is explained. Furthermore we give practical examples on determining the number of components (working with Eigenvalues and Scree plots), interpreting the component solution (working with component loadings), and rotating the component solution. Within the section on principal component analysis we will also address confirmative versus explorative analysis and the similarities and differences between component and factor analysis. At the end of the tutorial respondents will have had a thorough theoretical overview of the process of questionnaire development. Participants will be able to start developing their own questionnaires, and they will be able to correctly judge whether conclusion derived from questionnaire usage by others are reliable and valid. Finally, participants will gain experience in analyzing questionnaire data to evaluate both its reliability and validity. (See http://www.shortadress.com/interact)